Papers by Sagar Gubbi Venkatesh
UGIF-DataSet: A New Dataset for Cross-lingual, Cross-modal Sequential actions on the UI (2024.findings-naacl)
Copied to clipboard
| Challenge: | Identifying the right help document, understanding instructions from the document, and using them to resolve the issue at hand is challenging. |
| Approach: | They propose to use help documents to create step-by-step tutorials overlaid on the phone UI to overcome challenges in retrieval, parsing, and grounding in multilingual-multimodal setting. |
| Outcome: | The proposed dataset contains 4,184 tasks across 8 languages and shows that the end-to-end completion rate drops from 48% in English to 32% for other languages. |